Add 500 HTTP Error to retry list #2567
Conversation
|
hI @farzadab, thanks for the PR. |
|
My bad. Done. |
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
Wauplin
left a comment
Wauplin
left a comment
There was a problem hiding this comment.
Thanks! Looking good now :)
For the record, we've discussed this change internally (private slack). We had some concerns that retrying on HTTP 500 without knowing the root cause is clunky (can be transient error or not) but since:
- we retry only up to 5 times (so ~1min)
- and only when using
hf_file_systemso basically when streaming a dataset from the Hub
Then it's fine do to it. The cost of breaking a training process because of transient HTTP 500 is higher than when breaking a download process when instantiating a model once.
All of this to say, it's good to merge! 🤗
3 participants
As reported in #2559, reading from HF Hub datasets can sometimes lead to 500 Internal Server Errors, which even if they're rare, can cause issues on large training runs.
The source of this error is not yet identified, but a simple fix seems to be to add the missing 500 error code to the retry list.